System and method for efficient large-scale data processing

ABSTRACT

A large-scale data processing system and method includes one or more application-independent map modules configured to read input data and to apply at least one application-specific map operation to the input data to produce intermediate data values, wherein the map operation is automatically parallelized across multiple processors in the parallel processing environment. A plurality of intermediate data structures are used to store the intermediate data values. One or more application-independent reduce modules are configured to retrieve the intermediate data values and to apply at least one application-specific reduce operation to the intermediate data values to provide output data.

TECHNICAL FIELD

The disclosed embodiments relate generally to data processing systemsand methods, and in particular to a framework for simplifyinglarge-scale data processing.

BACKGROUND

Large-scale data processing involves extracting data of interest fromraw data in one or more datasets and processing it into a useful dataproduct. The implementation of large-scale data processing in a paralleland distributed processing environment typically includes thedistribution of data and computations among multiple disks andprocessors to make efficient use of aggregate storage space andcomputing power.

Various functional languages (e.g., LISP™) and systems provideapplication programmers with tools for querying and manipulating largedatasets. These conventional languages and systems, however, fail toprovide support for automatically parallelizing these operations acrossmultiple processors in a distributed and parallel processingenvironment. Nor do these languages and systems automatically handlesystem faults (e.g., processor failures) and I/O scheduling.

SUMMARY

A system and method for large-scale data processing includes operationsfor automatically handling programming details associated withparallelization, distribution, and fault-recovery. In some embodiments,application programmers can process large amounts of data by specifyingmap and reduce operations. The map operations retrieve data (e.g.,key/value pairs) from input data files and produce intermediate datavalues in accordance with the mapping operations. The reduce operationsmerge or otherwise combine the intermediate data values in accordancewith the reduce operations (e.g., combining intermediate values thatshare the same key). In some embodiments, the system and methods use amaster process to manage tasks and one or more local databases to reducenetwork traffic and file system (FS) reads.

In some embodiments, a system for large-scale processing of data in aparallel processing environment includes one or more map modulesconfigured to read input data and to apply at least oneapplication-specific map operation to the input data to produceintermediate data values. An intermediate data structure stores theintermediate data values. The system also includes reduce modules, whichare configured to retrieve the intermediate data values from theintermediate data structure and to apply at least one user-specifiedreduce operation to the intermediate data values to provide output data.Preferably, the map and/or reduce operations are automaticallyparallelized across multiple processors in the parallel processingenvironment. The programs or instructions for handling parallelizationof the map and reduce operation are application independent. The inputdata and the intermediate data values can include key/value pairs andthe reduce operation can include combining intermediate data valueshaving the same key. The intermediate data structure can include one ormore intermediate data files coupled to each map module for storingintermediate data values. The map and reduce operations can beimplemented on different processors coupled to a distributed network.The output data can be written to a file system, which is accessible viathe distributed network.

In some embodiments, a system for large-scale processing of data in aparallel processing environment includes a set of interconnectedcomputing systems. At least one of the computing systems includes a setof application independent map modules configured for reading portionsof input files containing data, and for applying at least oneapplication-specific map operation to the data to produce intermediatekey-value pairs. The system also includes a set of applicationindependent reduce modules, which are configured to apply at least oneapplication-specific reduce operation to the intermediate key-valuepairs so as to combine intermediate values sharing the same key. In oneembodiment, the application independent map modules and applicationindependent reduce modules are both incorporated into a same process,sometimes called a worker process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a large-scale data processing model.

FIG. 2 is a block diagram of a large-scale data processing system.

FIG. 3 is a block diagram of a large-scale data processing system,including a master process for managing tasks.

FIG. 4 is a block diagram of a computer system for the data processingsystems shown in FIGS. 2 and 3.

FIG. 5 is a block diagram of a data distribution network for large-scaledata processing.

FIG. 6 is a flow diagram of an embodiment of a process for assigningtasks to processes.

FIG. 7A is a block diagram of an exemplary task status table.

FIG. 7B is a block diagram of an exemplary process status table.

DESCRIPTION OF EMBODIMENTS Large-Scale Data Processing Model

FIG. 1 is a block diagram of a large-scale data processing model 100.The model 100 generally includes mapping operations 102 and reductionoperations 104. The mapping operations 102 apply one or more mappingoperations to a set of input data α_(i) (e.g., text files, records,logs, sorted maps, etc.) to provide a set of intermediate data valuesβ_(i). The reduction operations 104 apply one or more reductionoperations to the set of intermediate data values β_(i) to provide a setof output data φ_(i) (e.g., tables, sorted maps, record I/O, etc.). Insome embodiments, the mapping operations 102 are implemented by one ormore application-specific mapping functions, which map a set of inputdata α_(i) to a set of intermediate data values β_(i). The intermediatedata values β_(i) are stored in one or more intermediate datastructures. Some examples of intermediate data structures include,without limitation, files, buffers, histograms, count tables and anyother suitable data structure or device for storing digital information.The intermediate data values β_(i) are processed by the reductionoperations 104, which are implemented by one or moreapplication-specific reduction functions, which reduce the set ofintermediate data values β_(i) to a set of output data φ_(i).

Distributed Data Processing System

In order to explain the operation of the large scale data processingsystem, it is helpful to consider an exemplary distributed dataprocessing system in which the large scale data processing is performed.In general, the embodiments described here can be performed by a set ofinterconnected processors that are interconnected by one or morecommunication networks.

FIG. 5 is a block diagram of an exemplary distributed data processingsystem 500. It should be appreciated that the layout of the system 500is merely exemplary and the system 500 may take on any other suitablelayout or configuration. The system 500 is used to store data, performcomputational tasks, and transmit data between datacenters DC1-DC4. Thesystem may include any number of data centers DCx, and thus the numberof data centers shown in FIG. 5 is only exemplary. The system 500 mayinclude dedicated optical links or other dedicated communicationchannels, as well as supporting hardware such as modems, bridges,routers, switches, wireless antennas and towers, and the like. In someembodiments, the network 500 includes one or more wide area networks(WANs) as well as multiple local area networks (LANs). In someembodiments, the system 500 utilizes a private network, i.e., the systemand its interconnections are designed and operated exclusively for aparticular company or customer. Alternatively, a public network may beused.

Some of the datacenters DC1-DC4 may be located geographically close toeach other, and others may be located far from the other datacenters. Insome embodiments, each datacenter includes multiple racks. For example,datacenter 502 (DC1) includes multiple racks 508 a, . . . , 508 n. Theracks 508 can include frames or cabinets into which components aremounted. Each rack can include one or more processors (CPUs) 510. Forexample, the rack 508 a includes CPUs 510 a, . . . , 510 n (slaves 1-16)and the nth rack 506 n includes multiple CPUs 510 (CPUs 17-31). Theprocessors 510 can include data processors, network attached storagedevices, and other computer controlled devices. In some embodiments, atleast one of processors 510 operates as a master processor, and controlsthe scheduling and data distribution tasks performed throughout thenetwork 500. In some embodiments, one or more processors 510 may take onone or more roles, such as a master and/or slave. A rack can includestorage (e.g., one or more network attached disks) that is shared by theone or more processors 510.

In some embodiments, the processors 510 within each rack 508 areinterconnected to one another through a rack switch 506. Furthermore,all racks 508 within each datacenter 502 are also interconnected via adatacenter switch 504. As noted above, the present invention can beimplemented using other arrangements of multiple interconnectedprocessors.

Further details regarding the distributed network 500 of FIG. 5 can befound in U.S. patent application Ser. No. 10/613,626, entitled “Systemand Method For Data Distribution,” filed Jul. 3, 2003, which applicationis incorporated by reference herein in its entirety.

In another embodiment, the processors shown in FIG. 5 are replaced by asingle large-scale multiprocessor. In this embodiment, map and reduceoperations are automatically assigned to processes running on theprocessors of the large-scale multiprocessor.

Large-Scale Data Processing System I

FIG. 2 is a block diagram of a large-scale data processing system 200.The system 200 provides application programmers with anapplication-independent framework for writing data processing softwarethat can run in parallel across multiple different machines on adistributed network. The system 200 is typically a distributed systemhaving multiple processors, possibly including network attached storagenodes, that are interconnected by one or more communication networks.FIG. 2 provides a logical view of a system 200, which in someembodiments may be implemented on a system having the physical structureshown in FIG. 5. In one embodiment, the system 200 operates within asingle data center of the system 500 shown in FIG. 5, while in anotherembodiment, the system 200 operates over two or more data centers of thesystem 500.

As shown in FIG. 2, a set of input files 202 are processed by a firstset of processes 204, herein called map processes, to produce a set ofintermediate data, represented here by files 206. The intermediate data206 is processed by a second set of processes 208, herein called reduceprocesses, to produce output data 210. Generally each “map process” is aprocess configured (or configurable) to perform map functions and toexecute an application-specific map operator. Each “reduce process” is aprocess configured (or configurable) to perform reduce functions and toexecute an application-specific reduce operator. A control orsupervisory process, herein called the work queue master 214, controlsthe set of processing tasks. As described in more detail below, the workqueue master 214 determines how many map tasks to use, how many reducetasks to use, which processes and processors to use to perform thosetasks, where to store the intermediate data and output data, how torespond to any processing failures, and so on.

It should be noted that the work queue master 214 assigns tasks toprocesses, and that multiple processes may be executed by each of theprocessors in the group of processors that are available to do the workassigned by the work queue master 214. In the context of FIG. 5 or anyother multiple processor system, the set of processes controlled by thework queue master 214 may be a subset of the full set of processesexecuted by the system, and furthermore the set of processors availableto do the work assigned by the work queue master 214 may be fewer thanthe full set of processors in the system. Some of the resources of thesystem may be used for other tasks, such as tasks that generate theinput data 202, or that utilize the output data 210. However, in someembodiments, some or all of the tasks that generate the input data 202or utilize the output data 210 may also be controlled or supervised bythe work queue master 214. In addition, in some embodiments processorscan be added or removed from the processing system during the executionof a map-reduce operation. The work queue master 214 keeps track of theprocessors in the system and the available processes executing on thoseprocessors.

Application programmers are provided with a restricted set ofapplication-independent operators for reading input data and generatingoutput data. The operators invoke library functions that automaticallyhandle data partitioning, parallelization of computations, faulttolerance (e.g., recovering from process and machine failures) and I/Oscheduling. In some embodiments, to perform a specific data processingoperation on a set of input files, the only information that must beprovided by an application programmer provide is: informationidentifying of the input file(s), information identifying or specifyingthe output files to receive output data, and two application-specificdata processing operators, hereinafter referred to as map( ) and reduce(). Generally, the map( ) operator specifies how input data is to beprocessed to produce intermediate data and the reduce( ) operatorspecifies how the intermediate data values are to be merged or otherwisecombined. Note that the disclosed embodiments are not limited to anyparticular type or number of operators. Other types of operators (e.g.,data filters) can be provided, as needed, depending upon the system 200architecture and the data processing operations required to produce thedesired, application-specific results. In some embodiments, theapplication programmers provide a partition operator, in addition to themap( ) and reduce( ) operators. The partition( ) operator, specifies howthe intermediate data is to be partitioned over a set of intermediatefiles.

To perform large-scale data processing, a set of input files 202 aresplit into multiple data blocks 0, . . . , N-1 of either a specified orpredefined size (e.g., 64 MB). Alternately, in some embodiments theinput files 202 have a predefined maximum size (e.g., 1 GB), and theindividual files are the data blocks. A data block is a subset of datathat is retrieved during processing. In some embodiments, the datablocks are distributed across multiple storage devices (e.g., magneticor optical disks) in a data distribution network to fully utilize theaggregate storage space and disk bandwidth of the data processingsystem.

Referring to FIGS. 2 and 5, in some embodiments the input data files 202are stored in one or more data centers DC1-DC4. Ideally, the work queuemaster 214 assigns tasks to processors 510 in datacenters where theinput files are stored so as to minimize network traffic wheneverpossible. In some embodiments, the work queue master 214 uses input fileinformation received from a file system to determine the appropriateprocessor or process for executing a task, using a hierarchical decisionprocess. When a process in a processor in a datacenter DC1-DC4 is idle,it requests a task from the work queue master 214. The work queue master214 searches the input file information received from the file system(e.g., FS 446, FIG. 5), for an unprocessed data block on the machineassigned to process the task. If none are available, the work queuemaster 214 searches the file information for an unprocessed data blockon the same rack 508 as the machine assigned to process the task. Ifnone are available, the work queue master 214 searches the fileinformation for an unprocessed data block in the same datacenter as themachine assigned to process the task. If none are available, the workqueue master 214 will search for unprocessed blocks in otherdatacenters.

By using a hierarchical assignment scheme, data blocks can be processedquickly without requiring large volumes of data transfer traffic on thenetwork 500. This in turn allows more tasks to be performed withoutstraining the limits of the network 500.

Task Management

Referring again to FIG. 2, application programmers develop the map( )and/or reduce( ) operators, which are computer programs that processinput data and intermediate, respectively. In some embodiments theseoperators are compiled into binary files 212 suitable for use on aparticular processing platform. The binary files 202 are loaded into awork queue master module 214, which manages jobs submitted by users ofthe system 200. In some embodiments, the work queue master 214 loads (orcauses to be loaded) onto each process to which it allocates a map orreduce task, the library procedures, and the map( ) or reduce( )operator required to perform the task assigned to the process.

The work queue master 214, when it receives a request to process a setof data using a specified set application-specific map( ), reduce( )and, optionally, partition( ) operators, determines the number of maptasks and reduce tasks to be performed to process the input data. Thismay be based on the amount of input data to be processed. For example, ajob may include 10,000 map tasks and 10 reduce tasks. In someembodiments, the work queue master module generates a task status tablehaving entries representing all the tasks to be performed, and thenbegins assigning those tasks to idle processes. As noted above, tasksmay be allocated to idle processes based on a resource allocation scheme(e.g., priority, round-robin, weighted round-robin, etc.).

Process and Task Status Tracking

FIG. 6 is a flow diagram of an embodiment of a process 600 for assigningtasks to processes. Process 600 parallelizes a data processing task overas many processes as is consistent with the available computingresources. While the process 600 described below includes a number ofsteps that appear to occur in a specific order, it should be apparentthat the process 600 steps are not limited to any particular order, and,moreover, the process 600 can include more or fewer steps, which can beexecuted serially or in parallel (e.g., using parallel processors or amulti-threading environment). Further, it should noted that the steps oracts in process 600 are application-independent and are implementedusing modules or instructions that are application-independent. Only theactual map and reduce operators, which produce intermediate data valuesfrom the input data and that produce output data from the intermediatedata values, respectively, are application-specific. Theseapplication-specific operators are invoked by the map and reduce tasksassigned to processes in step 610. By making a clear boundary betweenthe application-independent aspects and application-specific aspects ofperforming a large scale data processing operation, theapplication-independent aspects can be optimized, thereby making theentire large scale data processing operation very efficient.

The process 600 begins by determining if there are tasks waiting to beassigned to a process (step 606). If there are no tasks waiting, thenthe process 600 waits for all the tasks to complete (step 604). If thereare tasks waiting, then the process 600 determines if there are any idleprocesses (step 608). If there are idle processes, then the process 600assigns a waiting task to an idle process (step 610) and returns to step606. If there are no idle processes, the process 600 waits for an idleprocess (step 614). Whenever a process completes a task, the processsends a corresponding message to the work queue master 214, whichupdates the process and task status tables (step 612). The work queuemaster 214 may then assign a new task to the idle process, if it has anyunassigned tasks waiting for processing resources. For reduce tasks, thework queue master 214 may defer assigning any particular reduce task toan idle process until such time that the intermediate data to beprocessed by the reduce task has, in fact, been generated by the maptasks. Some reduce tasks may be started long before the last of the maptasks are started if the intermediate data to be processed by thosereduce tasks is ready for reduce processing.

In some embodiments, whenever a process fails, which may be discoveredby the work queue master 214 using any of a number of known techniques,the work queue master 214 (A) determines what task was running in thefailed process, if any, (B) assigns that task to a new process, waitingif necessary until an idle process becomes available, and (C) updatesits process and task status tables accordingly. In some embodiments, thework queue master 214 may undertake remedial measures (step 602), suchas causing the failed process to be restarted or replaced by a newprocess. In some embodiments, the work queue master may further detectwhen such remedial measures fail and then update its process statustable to indicate such failures. In addition, in some embodiments, whena map task fails and is restarted in a new process, all processesexecuting reduce tasks are notified of the re-execution so that anyreduce task that has not already read the data produced by the failedprocess will read the data produced by the new process.

FIG. 7A shows an exemplary task status table for keeping track of thestatus of map and reduce tasks. In some embodiments, each task (e.g.,map, reduce) is assigned task ID, a status, a process, and one or moreinput files and output files. In some embodiments, the input files fieldmay specify a portion of an input file (e.g., where the portioncomprises a data block) to be processed by the task, or this field mayspecify portions of two of more input files. The status field indicatesthe current status of the task (e.g., waiting, in-progress, completed,or failed), which is being performed by the assigned process identifiedin the process field. The process retrieves data from one or more inputfiles (or the one or more input file portions) identified in the inputfile field and writes the results of the task ID to one or more outputfiles identified in the output file field. For example, in FIG. 7A, taskRed0000 is assigned to process P0033, which is still in progress. Theprocess P0033 retrieves data blocks from input file 12340 (e.g.,intermediate file A, FIG. 2) and writes the results of the task tooutput file 14000. In some embodiments, until a task is assigned to aprocess, the process field in the task status table indicates that noprocess has yet been assigned to perform that task. It should beapparent that there could be more or fewer fields in the task statustable than shown in FIG. 7A, such as multiple fields for identifyingoutput and input files.

FIG. 7B shows a process status table for keeping track of the status ofall the processes to which the work queue master 214 can assign tasks.In some embodiments, each process is assigned to a task and a location.In some embodiments, each process is permanently assigned to aparticular location (i.e., a particular processor). The status fieldindicates the current status of the process, which performs the assignedtask at the assigned location. For example, process P0010 is “Busy”performing task Map0103 on location CPU011. It should be apparent thatthere could be more or fewer field in the process status table thanshown in FIG. 7B, such as assigning multiple locations assigned to asingle task (e.g., parallel processing).

Map Phase

In some embodiments, the set of application-specific data processingoperations that the map( ) operator can perform is constrained. Forexample, in some embodiments, the map( ) operator may be required toprocess the input data one record at a time, proceeding monotonicallyfrom the first record to the last record in the data block beingprocessed. In some embodiments, the map( ) operator may be required togenerate its output data in the form of key/value pairs. Either the keyor value or both can comprise structured data, as long as the data canbe encoded into a string. For example, the key may have multiple parts,or the value may have multiple parts.

By requiring the map( ) operator's output to be in the form of key/valuepairs, the resulting intermediate data can be mapped to a set ofintermediate data files in accordance with a partition( ) operator. Anexemplary partition( ) operator may specify that all intermediate datais to be directed to an intermediate file corresponding to the value ofthe first byte of the key. Another exemplary partition( ) operator mayspecify that all intermediate data is to be directed to an intermediatefile corresponding to the value of the function “hash(Key) modulo N”,where N is a value specified by the application programmer and“hash(Key)” represents the value produced by applying a hash function tothe key of the key/value pairs in the intermediate data. In someembodiments, the partition operator is always a modulo function and theapplication programmer only specifies the modulus to be used by themodulo function. In one embodiment, the partition operator isautomatically selected by the work queue master 214, or by one of theapplication-independent library functions, discussed below.

In some embodiments, the data blocks 0, . . . , N-1 are automaticallyassigned to map tasks (executed by map processes 204-0, . . . , 204-N-1)in an application independent manner, by the work queue master 214. Inparticular, the work queue master 214 is configured to determine thenumber of data blocks to be processed, and to create a correspondingnumber of instances of the map process 204. Stated in another way, thework queue master 214 assigns a corresponding number of map tasks toprocesses, as suitable processes become available. Since the number ofmap tasks may exceed the number of processes available to the work queuemaster 214, the work queue master 214 will assign as many map tasks asit can to available processes, and will continue to assign the remainingmap tasks to processes as the processes complete previously assignedtasks and become available to take on new tasks. The work queue master214 uses the task status table and process status tables, describedabove, to coordinate its efforts.

Reduce Phase

Application independent reduce modules 208 read intermediate data values(e.g., key/value pairs) from the intermediate files 206. In someembodiments, each reduce module 208 reads from only one intermediatefile 206. The reduce modules 208 sort the intermediate data values,merge or otherwise combine sorted intermediate data values having thesame key and then write the key and combined values to one or moreoutput files 210. In some embodiments, the intermediate file 206 and theoutput files 210 are stored in a File System (FS), which is accessibleto other systems via a distributed network.

Software Implementation

In some embodiments, the map and reduce modules 204 and 208 areimplemented as user-defined objects with methods to carry outapplication-specific processing on data using known object orientedprogramming techniques. For example, a MapReduction base class can becreated that includes methods and data for counting the number of inputfiles that contain a particular term or pattern of terms, sorting theresults of the sort, eliminating duplicates in the sorted results andcounting the number of occurrences of the term. Application programmerscan derive other classes from the base class and instantiate the baseclass as an object in the application code to access its data andmethods.

Large-Scale Data Processing System II

While the system 200 provides good performance for many large-scale dataprocessing, the performance of the system 200 may diminish as the amountof data to be processed and thus the number of tasks increases. Forinstance, performance may be diminished when the size of the data blocksis decreased, thereby increasing the number of map tasks. Since theintermediate files 206 are stored in the FS, an increase in tasksresults in an increase in intermediate file access requests and anassociated increase in network traffic. Additionally, a single workqueue master 214 can only handle a limited number of task assignmentsper time period, beyond which the work queue master 214 begins to limitsystem performance. Increasing the size of those tasks to accommodateadditional jobs could result in load imbalances in the system 200. Theseperformance issues are addressed in the system 300, which is describedbelow with respect to FIG. 3.

FIG. 3 is a block diagram of a large-scale data processing system 300,including a master process 320 (sometimes called a supervisory process)for managing tasks. In system 300, one or more master processes 320assign one or more tasks to one or more worker processes 304 and 308. Insome embodiments, the master process 320 is a task itself (e.g., task 0)initiated by the work queue master module 314 and is responsible forassigning all other tasks (e.g., mapping and reducing tasks) to theworker processes 304, 308, in a master/slave type relationship. Theworker processes 304, 308 include two or more process threads, each ofwhich can be invoked based on the particular task assigned to it by themaster process 320. For example, each worker process 304 invokes a mapthread to handle an assigned map task and invokes a reduce thread tohandle an assigned reduce task. In one embodiment, the worker processes304, 308 include one or more additional threads. For example, a distinctthread may be used to receive remote procedure calls (e.g., from themaster process) and to coordinate work done by the other threads. Inanother example, a distinct thread may be used to handle remote readrequests received from other processors (i.e., peers) in the system.

In one embodiment, the number of worker processes is equal to the numberof machines available in the system 300 (i.e., one worker process permachine). In another embodiment, two or more worker processes are usedin each of the machines in the system 300. If a worker process fails,its task is reassigned to another worker process by the master process320. In some embodiments, the master process 320 or the work queuemaster 314 may undertake remedial measures to repair, restart or replacea failed worker process.

In some embodiments, when the work queue master 314 receives amap/reduce data processing job, the work queue master 314 allocates thejob to a master process 320. The master process 320 determines thenumber (M) of map tasks and the number (R) of reduce tasks to beperformed, and then makes a request to the work queue master 314 for M+Rprocesses (M+R+1, including the master process 320) to be allocated tothe map/reduce data processing job. The work queue master 314 respondsby assigning a number of processes to the job, and sends thatinformation to the master process 320, which will then manage theperformance of the data processing job. If the number of processesrequested exceeds the number of processes available, or otherwiseexceeds the number of processes that the work queue master 314 isallowed to assign to the job, the number of processes assigned to thejob will be less than the number requested.

In some embodiments, all R of the reduce tasks are all immediatelyassigned to processes, but the reduce tasks do not begin work (e.g., ondata sorting) until the master process 320 informs them that there areintermediate files ready for processing. In some embodiments, a singleworker process 304/308 can be assigned both a map task and a reducetask, simultaneously (with each being executed by a distinct processthread), and therefore assigning reduce tasks to processes at thebeginning of the job does not reduce the throughput of the system.

Map Phase

The division of input files 302 into data blocks 0, . . . , N-1, may behandled automatically by the application independent code. Alternately,the user may set an optional flag, or specify a parameter, so as tocontrol the size of the data blocks into which the input files aredivided. Furthermore, the input data may come from sources other thanfiles, such as a database or in-memory data structures.

The input data blocks 0, . . . , N-1, which may in some embodiments betreated as key/value pairs, are read by application independent workerprocesses 304-0, . . . , 304-N-1, as shown in FIG. 3. The input files302 can include a variety of data types typically used in dataprocessing systems, including without limitation text files, record I/O,sorted data structures (such as B-trees), tables and the like. Each ofthe worker processes 304 to which a map task has been assigned appliesthe application-specific map( ) operator to the respective input datablock so as generate intermediate data values. The intermediate datavalues are collected and written to one or more intermediate files 306,which are stored locally at the machine (e.g., in one or more localdatabases) in which the worker process 304 is executed. The intermediatefiles 306 are retained (i.e., they are persistent) until the reducephase completes. Note that in some embodiments, each of the intermediatefiles 306 receives output from only one worker process 304, as shown inFIG. 3. When a worker process 304 completes its assigned task, itinforms the master process 320 of the task status (e.g., complete orerror). If the task was successfully completed, the worker process'sstatus report is treated by the master process 320 as a request foranother task.

In some embodiments, if there are enough worker processes 304 that allthe intermediate values can be held in memory across the workerprocesses, then the system need not write any data to files on localdisks. This optimization reduces execution time for map-reduceoperations in which the number of worker processes is sufficient tohandle all the map tasks at once, and the amount of intermediate data issufficiently small to be kept in memory.

Application-Specific Combiner Function

In some cases, there is significant repetition in the intermediate keysproduced by each map task, and the application-specific Reduce functionis both commutative and associative. When all these conditions apply, aspecial optimization can be used to significantly reduce execution timeof the map-reduce task. An example of a situation in which theoptimization can be applied is a map-reduce operation for counting thenumber of occurrences of each distinct word in a large collection ofdocuments. In this example, the application-specific map function(sometimes called the map( ) operator elsewhere in this document)outputs a key/value pair for every word w in every document in thecollection, where the key/value pair is <w, 1>. The application-specificreduce function (sometimes called the reduce( ) operator elsewhere inthis document) for this example is:

input data is “values”;

int result=0; //initialize result to zero

for each v in values:

-   -   result+=ParseInt(v);

output: <key, result>

Each map task in this example will produce hundreds or thousands ofrecords of the form <word, 1>. The Reduce function simply adds up thecount values. To help conserve network bandwidth for map-reduceoperations that satisfy these properties, the user may provide anapplication-specific Combiner function or operator. The Combinerfunction is invoked with each unique intermediate key and a partial setof intermediate values for the key. This is similar to the Reducefunction, except that it gets executed at the end of each Map task bythe same machine and process that performed by Map task. The Combinerfunction partially summarizes the intermediate key/value pairs. In fact,when using a Combiner function, the same function is typically specifiedfor the Combiner and Reduce operations. The partial combining performedby the Combiner operation significantly speeds up certain classes ofMap-Reduce operations, in part by significantly reducing the amount ofinformation that must be conveyed from the processors that handle Maptasks to processors handling Reduce tasks, and in part by reducing thecomplexity and computation time required by the data sorting and Reducefunction performed by the Reduce tasks.

Reduce Phase

Application independent worker processes 308 which have been assignedreduce tasks read data from the locally stored intermediate files 306.In some embodiments, the master process 320 informs the worker processes308 where to find intermediate data files 306 and schedules readrequests for retrieving intermediate data values from the intermediatedata files 306. In some embodiments, each of the worker processes 308reads a corresponding one of the intermediate files 306 produced by allor a subset of the worker processes 304. For example, consider a systemin which each of the worker processes 304 assigned a map task outputs M(e.g., 100) intermediate files, which we will call Partion-1,j throughPartition-M,j, where j is an index identifying the map task thatproduced the intermediate files. The system will have 100 workerprocesses 308, Worker-1 to Worker-M, each of which reads a correspondingsubset of the intermediate files, Partition-p,j for all valid values of“j,” produced by the worker processes 304, where “p” indicates thepartition assigned to a particular worker process Worker-P (304) and “j”is an index identifying the map tasks that produced the intermediatefiles.

Each worker process 308 sorts the intermediate data values in the subsetof the intermediate files read by that worker process in accordance withthe key of the key/value pairs in the intermediate data. The sorting ofthe key/value pairs is an application independent function of the reducethreads in the worker processes 308. Each worker process 308 also mergesor otherwise combines the sorted intermediate data values having thesame key, and writes the key and combined values to one or more outputfiles 310. The merging or other combining operation performed on thesorted intermediate data is performed by an application-specific reduce() operator. In some embodiments, the output files 310 are stored in aFile System, which is accessible to other systems via a distributednetwork. When a worker process 308 completes its assigned reduce task,it informs the master process 320 of the task status (e.g., complete orerror). If the reduce task was completed successfully, the workerprocess's status report is treated by the master process 320 as arequest for another task. If the reduce task failed, the master process320 reassigns the reduce task to another worker process 308.

Recovering from Task and Processor Failures

In some embodiments, the master process 320 is configured to detect taskand processor failures. When a task failure is detected, the masterprocess 320 reassigns the task to another process. In some embodiments,the master process 320 redistributes the work of the failed task over alarger number of tasks so as to complete that task more quickly than bysimply re-executing the task on another process. The master processsubdivides the work assigned to the failed task to a plurality of newlymini-tasks, and then resumes normal operation by assigning themini-tasks to available processes. The number of mini-tasks may be apredefined number, such as a number between 8 and 32, or it may bedynamically determined based on the number of idle processes availableto the master process. In the case of a failed map task, division of thework assigned to the failed task means assigning smaller data blocks tothe mini-tasks. In the case of a failed reduce task, division of thework assigned to the failed task may mean assigning the data sortingportion of the reduce task to a larger number of worker processes,thereby performing a distributed sort and merge. The resulting sorteddata may, in some embodiments, be divided into a number of files orportions, each of which is then processed using the reduce( ) operatorto produce output data. By detecting such failures and taking theseremedial actions, the amount of delay in completing the entire dataprocessing operation is significantly reduced.

When a processor failure is detected by the master process 320, it maybe necessary to re-execute all the tasks that the failed processorcompleted as well as any tasks that were in process when the processorfailed, because the intermediate results produced by map tasks arestored locally, and the failure of the processor will in many cases makethose results unavailable. Using the status tables, described above, themaster process 320 determines all the tasks that ran on the processor,and also determines which of those tasks need to be re-executed (e.g.,because the results of the tasks are unavailable and are still needed).The master process 320 then updates its status tables to indicate thatthese identified tasks are waiting for assignment to worker tasks.Thereafter, re-execution of the identified tasks is automaticallyhandled using the processes and mechanisms described elsewhere in thisdocument.

In some embodiments, an additional mechanism, herein called backuptasks, is used to guard against task failures as well as task slowdowns. One of the main problems that lengthens the total time taken fora map-reduce operation to complete is the occurrence of “straggler”tasks or machines. A straggler is a process or machine that takes anunusually long time to complete one of the last few map or reduce tasksin the computation. Stragglers can arise for many reasons, includingboth hardware and software errors or conditions. When a large map-reduceoperation is divided into thousands of map and reduce tasks executed bythousands of processes, the risk of a straggler task occurring issignificant. The use of backup tasks, as described next, effectivelyguards against stragglers, without regard to the cause of the problemcausing a process or machine to run slowly. In these embodiments, themaster process determines when the map-reduce operation is close tocompletion. In one embodiment, the criteria for being close tocompletion is that the percentage of map tasks that have completed isabove a threshold. In another embodiment, the criteria for being closeto completion is that the percentage of map and reduce tasks, takentogether, that have completed is above a threshold. The threshold can beany reasonably number, such as 95, 98, or 99 percent, or any percentageabove 90 percent. Once the master process determines that the map-reduceoperation is close to completion, the master process schedules backupexecutions of all remaining tasks. These duplicate tasks may be calledbackup map tasks and backup reduce tasks. FIG. 7A shows an exemplarybackup task, Map103 b, in the task status table. Each task is marked ascompleted when either the primary or backup execution completes. Thismechanism obviously increases the computational resources, and thus insome embodiments the criteria for invoking this mechanism are selectedso as to increase the computational resources by no more than a fewpercent (e.g., five percent). The use of backup tasks significantlyreduces the time to complete large map-reduce operations, often by morethan twenty-five percent.

Master Process & Status Tables

The master process 320 is responsible for assigning tasks to the workerprocesses 304 and 308 and for tracking their status and output.Periodically, the master process 320 solicits a report from each workerprocess assigned a task to determine its task status. In someembodiments, the report can be solicited using a polling scheme (e.g.,round-robin). If the task status indicates that the worker process hasfailed, then the task is put back in the appropriate task queue to bereassigned to another worker process. In some embodiments, the masterprocess 320 maintains status tables 326 for managing tasks, as describedwith respect to FIGS. 7A and 7B.

In one embodiment in which more than one master process 320 is used, alocking mechanism is used to ensure that each of the entries of thestatus tables is modified by only one of the master processes at any onetime. Whenever a master process 320 attempts to assign a map or reducetask to a process, or perform any other management of a map or reducetask, the master process first acquires (or attempts to acquire) a lockon the corresponding status table entry. If the lock is refused, themaster process concludes that the map/reduce task is being managed byanother master process and therefore the master process looks foranother map/reduce task to manage. In another embodiment, the taskstatus table is divided into portions, with each master process beinggiven ownership of a corresponding portion of the task status table, andresponsibility for managing the map/reduce tasks in that portion of thetask status table. Each master process can read other portions of thetask status table, but only uses information in entries indicating thatthe corresponding task has been completed.

The system 300 provides several advantages over other systems andmethods by using one or more master processes to assign and managetasks, together with local databases to store intermediate resultsproduced by the tasks. For example, by distributing file reads overmultiple local databases more machines can be used to complete tasksfaster. Moreover, since smaller tasks are spread across many machines, amachine failure will result in less lost work and a reduction in thelatency introduced by such failure. For example, the FS load for system200 is O(M*R) file opens and the FS load for system 300 is O(M) inputfile opens +O(R) output file opens, where M is the number of map tasksand R is the number of reduce tasks. Thus, the system 200 requiressignificantly more file system file open operations than the system 300.

Computer System for Large-Scale Data Processing

FIG. 4 is a computer system 400 for the data processing systems 200 and300 shown in FIGS. 2 and 3. The computer system 400 generally includesone or more processing units (CPUs) 402, one or more network or othercommunications interfaces 410, memory 412, and one or more communicationbuses 414 for interconnecting these components. The system 400 mayoptionally include a user interface 404, for instance a display 406 anda keyboard 408. Memory 412 may include high speed random access memoryand may also include non-volatile memory, such as one or more magneticdisk storage devices. Memory 412 may include mass storage that isremotely located from the central processing unit(s) 402.

The memory 412 stores an operating system 416 (e.g., Linux or Unix), anetwork communication module 418, a system initialization module 420,application software 422 and a library 430. The operating system 416generally includes procedures for handling various basic system servicesand for performing hardware dependent tasks. The network communicationmodule 418 is used for connecting the system 400 to a file system (FS)446, servers or other computing devices via one or more communicationnetworks, such as the Internet, other wide area networks, local areanetworks, metropolitan area networks, and the like. The systeminitialization module 420 initializes other modules and data structuresstored in memory 414 required for the appropriate operation of thesystem 400. In some embodiments, the application software 422 includes amap operator 424, a reduce operator 426 and a partition operator 428,and the library 430 includes application-independent map functions 432,reduce functions 434, and partition functions 436. As discussed above,the application software 422 may also include a combiner operator 425when the map-reduce operation meets certain conditions. The functions,procedures or instructions in the library 430 handle the applicationindependent aspects of large scaled data processing jobs, while theapplication software 422 provides the application-specific functions forproducing output data. The application software 422 may include sourceprograms for the map, combiner, reduce and partition operators as wellas the corresponding compiled programs, represented by binary files 212and 312 in FIGS. 2 and 3, respectively.

One or more status tables 444 are also included to track tasks andprocesses, as described with respect to FIGS. 7A and 7B. In someembodiments, the computer system 400 includes worker processes 438,intermediate files 440, and one or more master process(es) 442. Theinteraction of worker processes 438 and master processes 442 weredescribed with respect to FIG. 3.

Referring to FIGS. 2, 3 and 4, an application programmer can create ascript or program using the application software 422, which includes oneor more operators 424, 426 and 428. The script or program is processedinto binary files 212, 312 and provided to the work queue master 214,314.

For the embodiment shown in FIG. 2, input files 202 are split intomultiple data blocks and assigned by the work queue master 214 toindividual, application independent map and reduce processes 204 and208. The processes 204 invoke map functions 432 to process the inputdata (e.g., counting the number of occurrences of a term) to provideintermediate data values. In some embodiments, the input data isstructured in the form of key-value pairs. The partition function 436partitions the map output into one or more intermediate files 440, whichare stored on the FS 446. The intermediate data values are processed bythe map and reduce processes 204 and 208, which invoke reduce functions208 for sorting and combining intermediate data values having the samekey, and for storing the key and values in one or more output files 210located on the FS 446. The work queue master 214 manages the map andreduce processes 204 and 208 with the assistance of status tables 444,as described with respect to FIGS. 7A and 7B.

For the embodiment shown in FIG. 3, input files 302 are split intomultiple data blocks and assigned by the master process 442 toindividual, application independent worker processes 438. The workerprocesses 438 invoke map functions 432 for operating on blocks of inputdata (e.g., counting the number of occurrences of a term) to provideintermediate data values. The partition function 436 partitions the mapoutput into one or more intermediate files 440, which are stored locallyin memory 412. The intermediate data values are processed by applicationindependent worker processes 438, which invoke reduce functions 434 forsorting and combining intermediate data values having the same key, andfor storing the resulting output data in one or more output files 310located on the file system 446. The master process 442 manages theworker processes 436 with the assistance of status tables 444, asdescribed with respect to FIGS. 7A and 7B.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A system for large-scale processing of data, comprising: a pluralityof processes executing on a plurality of interconnected processors; theplurality of processes including a master process, for coordinating adata processing job for processing a set of input data, and workerprocesses; the master process, in response to a request to perform thedata processing job, assigning input data blocks of the set of inputdata to respective ones of the worker processes; each of a firstplurality of the worker processes including an application-independentmap module for retrieving a respective input data block assigned to theworker process by the master process and applying anapplication-specific map operation to the respective input data block toproduce intermediate data values, wherein at least a subset of theintermediate data values each comprises a key/value pair, and wherein atleast two of the first plurality of the worker processes operatesimultaneously so as to perform the application-specific map operationin parallel on distinct, respective input data blocks; a partitionoperator for processing the produced intermediate data values to producea plurality of intermediate data sets, wherein each respectiveintermediate data set includes all key/value pairs for a distinct set ofrespective keys, and wherein at least one of the respective intermediatedata sets includes respective ones of the key/value pairs produced by aplurality of the first plurality of the worker processes; and each of asecond plurality of the worker processes including anapplication-independent reduce module for retrieving data, the retrieveddata comprising at least a subset of the key/value pairs from arespective intermediate data set of the plurality of intermediate datasets and applying an application-specific reduce operation to theretrieved data to produce final output data corresponding to thedistinct set of respective keys in the respective intermediate data setof the plurality of intermediate data sets, and wherein at least two ofthe second plurality of the worker processes operate simultaneously soas to perform the application-specific reduce operation in parallel onmultiple respective subsets of the produced intermediate data values. 2.The system of claim 1, wherein each of the worker processes includes amap process thread and a reduce process thread, the map process threadconfigured to execute the application-independent map module and thereduce process thread configured to execute the application-independentreduce module.
 3. The system of claim 1, wherein the master process isconfigured to automatically determine a number of distinct map tasks anda number of distinct reduce tasks to perform the data processing job,and to automatically assign the distinct map tasks and the distinctreduce tasks to the worker processes in accordance with availability ofthe worker processes executing on the interconnected processors suchthat some of the distinct map tasks and some of the distinct reducetasks are assigned to worker processes later, during performance of thedata processing job, than other ones of the distinct map tasks and thedistinct reduce tasks.
 4. The system of claim 3, wherein the number ofdistinct map tasks exceeds in number the first plurality of the workerprocesses to which the master process can assign the distinct map tasks,and wherein the master process maintains status information with respectto the distinct map tasks awaiting assignment to one of the firstplurality of worker processes.
 5. The system of claim 3, wherein themaster process is configured to maintain a task status table, denotingfor each of the distinct map tasks and each of the distinct reduce tasksone of the worker processes, if any, to which each of the distinct maptasks or each of the distinct reduce tasks has been assigned, and astatus of each of the distinct map tasks or each of the distinct reducetasks.
 6. The system of claim 5, wherein the master process is furtherconfigured to maintain a process status tables, denoting which of theworker processes has been assigned one of the distinct map tasks, andwhich of the worker processes has been assigned one of the distinctreduce tasks, and a status of each of the worker processes.
 7. Thesystem of claim 5, wherein the application-specific map operationincludes an application-specific combiner operation for combininginitial data values produced by the application-specific map operationso as to produce the intermediate data values.
 8. The system of claim 5,wherein the application-specific map operation includes anapplication-specific combiner operation for combining initial datavalues produced by the application-specific map operation having sharedkeys so as to produce the intermediate data values.
 9. A method ofperforming a large-scale data processing job, comprising: executing aplurality of processes on a plurality of interconnected processors, theplurality of processes including a master process for coordinating thelarge-scale data processing job for processing a set of input data, andworker processes; in the master process, in response to a request toperform the large-scale data processing job, assigning input data blocksof the set of input data to respective ones of the worker processes; ineach of a first plurality of the worker processes, executing anapplication-independent map module to retrieve a respective input datablock assigned to the worker process by the master process and to applyan application-specific map operation to the respective input data blockto produce intermediate data values, wherein at least a subset of theintermediate data values each comprises a key/value pair, and wherein atleast two of the first plurality of the worker processes operatesimultaneously so as to perform the application-specific map operationin parallel on distinct, respective input data blocks; using a partitionoperator to process the produced intermediate data values to produce aplurality of intermediate data sets, wherein each respectiveintermediate data set includes all key/value pairs for a distinct set ofrespective keys, and wherein at least one of the respective intermediatedata sets includes respective ones of the key/value pairs produced by aplurality of the first plurality of the worker processes; and in each ofa second plurality of the worker processes, executing anapplication-independent reduce module to retrieve data, the retrieveddata comprising at least a subset of the key/value pairs from arespective intermediate data set of the plurality of intermediate datasets and applying an application-specific reduce operation to theretrieved data to produce final output data corresponding to thedistinct set of respective keys in the respective intermediate data setof the plurality of intermediate data sets, and wherein at least two ofthe second plurality of the worker processes operate simultaneously soas to perform the application-specific reduce operation in parallel onmultiple respective subsets of the produced intermediate data values.10. The method of claim 9, wherein each of the worker processes includesa map process thread and a reduce process thread, the map process forexecuting the application-independent map module and the reduce processthread for executing the application-independent reduce module.
 11. Themethod of claim 9, wherein the master process maintains a task statustable, denoting for each of the distinct map tasks and each of thedistinct reduce tasks the respective ones of the worker processes, ifany, to which each of the distinct map tasks or each of the distinctreduce tasks has been assigned, and a status of each of the distinct maptasks or each of the distinct reduce tasks.
 12. The method of claim 11,wherein the master process maintains a process status table, denotingwhich of the worker processes has been assigned one of the distinct maptasks, and which of the worker processes has been assigned one of thedistinct reduce tasks, and a status of each of the worker processes. 13.The method of claim 9, wherein the application-specific map operationincludes an application-specific combiner operation for combininginitial data values produced by the application-specific map operationso as to produce the intermediate data values.
 14. The method of claim9, wherein the application-specific map operation includes anapplication-specific combiner operation for combining initial datavalues produced by the application-specific map operation having sharedkeys so as to produce the intermediate data values.
 15. The method ofclaim 9, including, in the master process, automatically determining anumber of distinct map tasks and a number of distinct reduce tasks toperform the data processing job, and automatically assigning thedistinct map tasks and the distinct reduce tasks to the worker processesin accordance with availability of the worker processes such that someof the distinct map tasks and some of the distinct reduce tasks areassigned to worker processes later, during performance of thelarge-scale data processing job, than other ones of the distinct maptasks and the distinct reduce tasks.
 16. The method of claim 15, whereinthe number of distinct map tasks exceeds in number the first pluralityof the worker processes to which the master process can assign thedistinct map tasks, and wherein the method includes, in the masterprocess, maintaining status information with respect to the distinct maptasks awaiting assignment to one of the first plurality of workerprocesses.